Tag
21 articles
Google has released LiteRT.js, a JavaScript binding of its LiteRT library that enables running .tflite models in browsers via WebGPU, offering up to 60x performance gains over CPU-only execution.
OpenAI has unveiled its first custom AI processor, Jalapeño, developed in partnership with Broadcom. The chip is designed for AI inference tasks and marks a move toward vertical integration in the AI industry.
Learn how to create and run a simple AI inference example, understanding the core concepts behind AI model deployment that companies like Baseten are building upon.
Xiaomi's MiMo team, with TileRT, has achieved over 1000 tokens per second on a 1-trillion-parameter model using a single 8-GPU commodity node, marking a significant leap in LLM inference performance.
Perplexity AI introduces a hybrid local-server inference orchestrator that automatically routes AI tasks between on-device and cloud models, enhancing both performance and privacy.
NVIDIA introduces Dynamo Snapshot, a CRIU-based system that accelerates AI inference on Kubernetes by enabling fast startup and restoration of vLLM workers.
Perplexity AI has introduced an intelligent system that dynamically splits AI workloads between local PCs and cloud servers, optimizing performance and cost.
AI chip startup Groq is raising $650 million in internal funding as it pivots from hardware to focus more on AI inference, the process of refining how AI models respond to user prompts.
Google introduces the Coral Board, a compact single-board computer that runs Gemma 3 locally, enabling on-device AI inference for edge computing applications.
Together AI open-sources OSCAR, an attention-aware 2-bit KV cache quantization system that significantly reduces memory usage and improves decoding speed for long-context LLMs.
This article explains how the surge in demand for Apple's Mac mini on eBay reflects the growing need for local AI inference hardware, demonstrating the intersection of AI model complexity, hardware scarcity, and market dynamics in the emerging AI ecosystem.
Learn how Google and NVIDIA are making AI inference cheaper and faster through new hardware and software integration. This breakthrough could make AI more accessible to businesses and improve everyday applications.
